Interactive Audio-Tactile Annotation of 3D Point Clouds for Robotic Manipulation

A major limitation of 3D point cloud-based representations is that they often fail to capture an object’s functional features – a 3D model of a stapler does not specify where the stapler should be pressed. For many tools, successful manipulation requires that the robot detects the specific location that needs to be actuated (e.g., a trigger or a button) to turn on the tool. Yet, to date, virtually all methods for robotic tool use rely on 3D object models and require the programmer to specify what part of the object needs to be actuated. Recently, interactive perception approaches have been proposed for problems ranging from discovering the degrees of freedom of an object [1] to autonomous learning of manipulation skills [2]. Inspired by these lines of work, we propose a method for a robot to interactively detect the functional parts of tools using tactile and auditory percepts coupled with exploratory behaviors. The approach consists of detecting auditory and tactile events while manipulating an object and mapping those events onto the object’s 3D model. The result is a novel visual-audio-tactile point cloud representation which not only captures the object’s shape, but also encodes its functional components (e.g., buttons) that afford actuation.